Relational Memory-Augmented Language Models

Abstract We present a memory-augmented approach to condition an autoregressive language model on knowledge graph. represent the graph as collection of relation triples and retrieve relevant relations for given context improve text generation. Experiments WikiText-103, WMT19, enwik8 English datasets demonstrate that our produces better in terms perplexity bits per character. also show relational memory improves coherence, is complementary token-based memory, enables causal interventions. Our provides simple yet effective way combine more coherent logical